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ABSTRACT 



In 1985, Arizona initiated a 5-year pilot test of the 



Career Ladder Teacher Incentive and Development Program. The program 
provided alternate avenues for teachers to advance other than by 
accumulating years of experience and college credits. Data were 
collected annually through the Career Ladder Perception Assessment 
Scale Survey, which evaluated the program in terms of recruiting, 
retaining and motivating high-quality teachers who impact positively 
on student achievement. Results of a series of factor analyses 
incorporated into nine tables present data as follows: (1) Summary 
Statistics: Principal Factor Method; (2) Factor I: Program Reform and 
Accountability; (3) Factor 2: Communication and Emotional Health; (A) 
Factor 3: Evaluation and Placement; (5) Factor A: Psychological 
Self-Actuali.^.ation; (6) Factor 5: Peer Evaluation; (7) Factor 6: 
Program Support; (8) Factor 7: Teacher Input and Advancement 
Opportunities; and (9) Cronbach^s Alpha Values for Seven Extracted 
Factors. Seventy references, a series of legislative reports and 
pubiicanions related to the project, and a copy of the Career Ladder 
Perception Assessment Scale Survey complete the document. (LL) 
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BESTWYfimflE 



Validity of Key Factors Within the 
Assessment Instrumentation Used for the 
Evaluation of the Arizona Career Ladders Program 

Overview 

In 1985, Arizona initiated the five-year pilot test of the Career Ladder Teacher 
Incentive and Development Program. This program provided alternate avenues for 
teachers to advance other than solely by accumulating years of experience and college 
credits. Arizona's program was unique regarding its explicit recognition of local 
diversity. Participating districts were granted considerable autonomy regarding how they 
developed their local career ladder plans and requirements. 

The pilot project resulted in documented gains in student academic achievement, 
teacher performance and district readiness levels (Packard and Dereshiwsky, 1989, 
November 3; Brandt, 1990). As a result, the Arizona State Legislature has extended 
funding for the program beyond the pilot test period. Additionally, the program is 
expanding in scope; a phase-in of additional applicant-districts has been approved. 

Evaluation and Poli cy Research 

Evaluation and policy research are methodologies which follow the scientific 
approach to knowledge, and they usually focus on a problem of assessing the impact of 
some organizational program or governmental policy. Program developers in education 
and govemment have recently been reaping the benefits of applying these techniques to 
determine the value of programs in meeting intent or purposes. "Performance analysis" 
has been applied for some time in the areas of business and manufacturing, but only 
recently have governments and educational agencies adopted sim'^lar systematic 
techniques of demonstrating objective evidence for accountability. In the past, most 
social or governmental programs were continued on the basis of special interest groups, 
power politics, or some type or level of authority; but litde information was available 
with which to make sound decisions based on objective data. 

In 1984, Arizona was fortunate to have a legislature who did not follow the 
procedures used by other states who were on the reform movement bandwagon related to 
"career ladders" programs. Unlike most other states, a key aspect of the Arizona 
legislation was to provide an outside evaluator to assess the worth of the program in 
recruiting, retaining and motivating high-quality teachers who could make a difference in 

1 



student learning and development (or achievement). The Center for Excellence in 
Education, at Northern Arizona University in Flagstaff, was selected to provide the 
research and reports to the State Legislature and Career Ladder Task Force over the five- 
(5) year pilot-test Arizona Career Ladder Evaluation Project (S.B. 1521, 1985). 

In addition to the legislative reports, a series of published research papers resulted 
from the long-range pilot-test program. Please see a listing of these reports and 
publications in the reference section of this document (Packard, Dereshiwsky, Bierlein 
and others, 1985-1992). 

Study Design and Procednrp,.s 

The Joint Legislative Committee on Career Ladders required that the project be 
evaluated. Data were collected annually ui-.,g an instrument entitled the Career Ladder 
Perception A??g$sment Sea ls (CLPAS) Survey (Packard, Bierlein, Aleamoni and 
Healmstadter, 1986). This questionnaire (See Exhibit A for a copy of the questionnaire) 
consisted of 53 Likert-scaled items (strongly disagree to strongly agree, with a not- 
applicable option) in the following areas: 

1 . general Career Ladder concep ts; 

2. staff development concepts; 

3. teacher evaluation concepts; 

4. peer evaluation concepts; 

5. Career Ladder placement concepts; 

6. organizational climate. 

The paper presents the results of a series of factor analyses conducted on the 
preceding concept areas (subscales) of the CLPAS. The analysis was accomplished using 
the principal factor method of factor extraction (Dillon & Goldstein, 1984; Tabachnik & 
Fidell, 1989. Kim & Mueller (1978) have also provided a succinct description of factor 
analysis procedures in one of the Sage series documents. ). Extracted factors were then 
subjected to varimax rotation. 

The database consists of the 1989-90 responses to this survey. Subjects were 
selected using a probability proportional to size (PPS) random sample stratified by 
panicipating district. A total of 2,880 usable (at least partially completed) responses were 
received. 

Resuhs 

Table 1 (p. 3) presents the factor-analytic results. The related seven (7) factorial 
constructs are as follows: (1) Program Refomi and Accountability; (2) Communication 
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and Emotional Health; (3) Evaluation and Placement; (4) Psychological Self- 
Actualization, (5) Peer Evaluation; (6) Program Support Factors, and (7) Teacher Input 
and Advancement Opportunities, 

As shown in Table 1, four factors had eigenvalues greater Tnan or equal to LOO. 
These factors accounted for 59.1% of the total variance. In addition, factor solution 
extracted a significant amount of covariance, as evidenced the application of Bartlett's 
test of sphericity (14,953,74; p-value=O.O0). The KMO index o.^ ^^ar::ipling adequacy was 
0.97, also indicative of a robust factor solution. 



Table 1. 

Summary Statistics: Principal Factor Method 



Factor 


Eigenvalue 


Percent of Variance 
Extracted 


I .imulative Percent 
i of Variance 
Exti'acted 


Program Reform 
and Accountability 


25.986 


49.0 


49.0 


Communication and 
Emotional Health 


2.807 


5.3 


J4.3 


Evaluation and 
Placement 


1.467 


2.8 


57.1 


Psychological Self- 
Actualization 


1.077 


2.0 


59.1 


Peer Evaluation 


0.770 


1.5 


6'^ ^ 


Program Support 


0.718 


1.4 


02.J 


Teacher Input and 

Advancement 

Opportunities 


0.590 


1.1 


63.1 



Bartlett's Test of Sphericity: 14,953.74 
Significance: 0.00 
KMO Index of Sampling Adequacy: 0.97 

Tables 2 through 8 (pgs. 4 - 8) contain the survey items which loaded 
significantly (loadings of 0.50 or greater in absolute value) on each of the seven extracted 
factors. A brief interpretation of each factor will be provided along with the 
corresponding table. 



Table 2 (p. 4) depicts key items which relate to the "program reform and 
accountability" factor. Concepts which were of central importance to the national reform 
movement loaded to a significant degree. These factors included: (1) focus on improved 
instruction and student outcomes performance, (2) recruitment, retention and motivation 
of high-quality teachers, (3) favorable time/benefit ratio, (4) cooperative interpersonal 
relationships, and (5) challenging criteria and clarity of performance goals. 



Table 2. 

Factor 1: Program Reform and Accountability 



Survey Item Number 


Item Content 


Factor Loading 


GEN16 


Improved instruction 


0.810 


UEN17 


Student progress improved 


0.806 


GEN 15 


Compet ;nt teachers retained 


0.793 


GENU 


Comoetent tearhers 
recruited 


0 781 


GEN 19 


ImDroved teacher morale 


0 740 


GEN20 


Improved perceived 
professional status 


0.729 


GEN22 


Intrinsic rewards available 


0.720 


TEV32 


Time spent is worth benefits 
gained 


0.642 


GEN 18 


Teacher cooperation 
encouraged 


0.635 


TEV34 


Student outcomes reflect 
performance 


0.550 


TEV33 


Proper emphasis placed on 
achievement 


0.523 


CLP44 


Challenging CLP criteria 


0.523 


CLP47 


Outside advancement 
opportunities 


0.480 


GEN23 


Goals are clearly 
communicated 


0.472 


CLP53 


Teachers have input into 
CLP revisions 


0.470 


PEV40 


Staff cooperation is 
encouraged 


0.460 


CLP51 


Top responsibilities are 
appropriate 


0.456 


CLP43 


Can stay at same CLP level 


0.452 


CLP48 


Teachers are involved in 
CLP development 


0.443 


TEV28 


Performance goals are 
clearly defined 


0.419 
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GEN: General Career Ladder Concepts; 

CLP: Career Ladder Placement Concepts; 

TEV: Teacher Evaluation Concepts; 

PEV: Peer Evaluation Concepts. 

Table 3 (p. 5) shows the imponance of items related to the "communication and 
emotional health" factor. Key concepts included: (1) communication type and level, 
cooperation and social interrelationships and positive psychological/emotional feelings, 
(2) clarity of organizational goals, purpose, progress and criteria for advancement, and (3) 
importance of good leadership models. 

Table 3. 

Factor 2: Communication and Emotional Health 



Survey Item Number 


Item Content 


Factor Loading 


CLIM66 


Communication level feels 
good 


0.798 


CLIM65 


Strong social network 


0.703 


CLIM59 


Cooperative work 
environment 


0.634 


CLIM64 


Goals are clearly 
communicated 


0.602 


CLIM60 


Good leadership models 


0.600 


CLIM61 


Stress-free environment 


0.501 


CLIM58 


Get feedback on progress 


0.47" 


CLP42 


Advancement criteria are 
understood 


0.400 



CLIM: Organizational Climate Concepts; 

CLP: Career Ladder Placement Concepts. 

Table 4 (p. 6) establishes the importance of "evaluation and placement" concepts. 
Those items include the following: (1) qualification, training and fairness of peer and 
administrative teacher evaluators, and (2) adequacy of support for teaching skill 
improvement. 
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Table 4. 



Factor 3: Evaluation and Placement 



Survey Item Number 


Item Content 


Factor Loading 


STAFF26 


Peer evaluators are well 
trained 


0.663 


PEV36 


Weil-trained evaluators 


0.661 


PEV35 


Peer evaluators are chosen 
for top qualification': 


0.571 


STAFr25 


Administrators are v/ell 
trained 


0.543 


'l'HV29 


Administrators evaluate 
fairly 


0.542 


TEV31 


Evaluation time is sufficient 


0.499 


TEV30 


Evaluation procedures are 
consistent 


0.471 


STAFF27 


Adequate teacher skills 
resources 


0.430 



STAFF: Staff Development Concepts; 
TEV: Teacher Evaluation Concepts; 

PEV: Peer Evaluation Concepts. 

Table 5 (p. 6) depicts concepts related to "psychological/emotional self- 
actualization" as follows: (1) feelings of purpose, success, imponance, security, 
belonging and cooperation, and (2) feelings of being rewai'ded for a job well done. 

^^ble 5. 

Factor 4: Psychological Self-Actualization 



Survey Item Number 


Item Content 


Factor Loading 


CL1M57 


Work has clear purpose 


0.785 


CLIM55 


Feel successful In job 


0.748 


CLIM62 


Job function is important 


0.524 


CLIM54 


Feel I belong 


0.521 


CLIM63 


Feel secure in job status 


0.520 


CL1M56 


Feel rewarded in job 


0.477 


CLIM59 


Cooperative work 
environment 


0.418 



CUM: Organizational Climate Concepts. 
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The remaining three extracted factors had eigenvalues of less than one (1) and 
accou'^ted for 4.0% of the total variance. In addition, the final extracted factor (Table 8, 
p. 8) consisted of "redundant" items (e.g., survey questions that had already loaded on 
previously extracted factors). Tables 6, 7 and 8 (pgs. 7 - 8) show those factors as follows: 
(1) teacher program input, (2) evaluation system, (3) adequacy of suppon for materials 
development and inservice, and (4) placement and advancement opportunities. 



Table 6, 
Factor 5: Peer Evaluation 



Survey Item Number 


Item Content 


Factor Loading 


PEV37 




Teachers have enough 
selection input 


0.670 


PEV38 




Peer evaluation is formative 


0.573 


PEV40 




Peer evaluation is 
summative 


0.463 



PE V : Peer E val uation Concepts . 



Table 7, 
Factor 6: Program Support 



Survey Item Number 


Item Content 


Factor Loading 


CLP46 


Adequate assistance with 
materials 


0.589 


CLP45 


Evaluation materials 
standards exist 


0.529 


GEN23 


Goals are clearly 
communicated 


0.498 


CLP42 


Advancement criteria are 
understood 


0.464 


STAFF24 


Adequate inservice received 


0.408 



GEN: General Career Ladder Concepts; 

CLP: Career Ladder Placement Concepts; 

STAFF: Staff Development Concepts. 
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Table 8. 



Factor 7: Teacher Input and Advancement Opportunities 



Suivev Item Number 


Item Content 


Factor Loading 


CLP48 


Teachers are involved in 
CLP placement 


0.454 


CLP47 


Outside advancement 
opportunities 


0.405 


CLP53 


Teachers have input into 
CLP revisions 


0,404 



CLP: Career Ladder Placement Concepts. 

As a final step, Cronbach's alpha reliability coefficients, along with per-item 
deletion values, were calculated for the individual survey items loading "highly" 
(loadings of 0.50 or greater in absolute value) within each factor. (Both the standardized 
and unstandardized values are shown; since the survey items were identically scaled, 
these should be equivalent except for rounding error.) 

Table 9 (p. 8) displays these results. As can be seen ft'om this table, the four 
factors with eigenvalues of 1.00 or greater also yielded Cronbach's alpha coefficients 
greater than or equal to 0.85. All of the seven extracted factors yielded Cronbach's alpha 
values above the "marginally acceptable" 0.60 value. In addition, the per-item deletion 
did not result in any notable improvement in calculated alpha values. 

Table 9, 

Cronbach's Alpha Values for Seven Extracted Factors 



Factor 


Cronbach's Alpha 


Standardized Cronbach's 
Alpha 


Program Reform and 
Accountabilitv 


0.9653 


0.9650 


Communication and 
Emotional Health 


0.9028 


C.9043 


Evaluation and Placement 


0.9303 


0.9306 


Psychological Self- 
Actualization 


0.8596 


0.8640 


Peer Evaluation 


0.6981 


0.6986 


Program Support Factors 


0.8588 


0.8589 


Teacher Input and 
Advancement Opportunities 


0.6984 


0.6981 
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Cross- Validity to Related Policy 

In conclusion, the initial survey instrumentation and related policy results of the 
cai'cer ladder programs in Arizona have been relatively positive and closely associated. 
In pari, this is due to the use of systematic and objective evaluation research techniques 
which allowed decision rnalcers to have relevant data with which to continue programs 
and refine legislation. There is a sirong and direct relation between factors that loaded on 
the evaluation instrumentation and final components of legislation that supported the 
value or worth of this reform program. Therefore, the issue of validity of insti'umentanon 
is fairly well being borne out by both the factor-analytic (quantitative) and field-based 
historical (qualitative) results. 

The program intent was, and still is, to recruit, retain, and motivate high-quality 
teachers and result in a positive impact on student achievement. Specific factors v/hich 
significantly loaded, and are an imponant part of the restructuring presently going on 
with current career ladder programs, are as follows: (1) program reform for improved 
instruction and student outcomes performance, recruitment, retention and monvation of 
high-quality teachers, and positive nme/l)enefits ratio; (2) communication factors related 
to a cooperative organizational climate, psychological environment, clarity of purpose, 
good leadership models, and support resources; (3) emotional health and psychological 
support factors of clear purpose, feelings of job importance, success, belonging, security, 
reinforcement (reward), and cooperadve environment; (4) the imponance of a formative 
and summative evaluation system; and (5) other program support factors which relate to 
teacher input and advancement opportunities. 

The continuing refinement of each of the imponant cc ^Donents of the cai'eer 
ladder refonn movement is an ongoing process. Beyond the original pilot-test districts, 
several new ones are being accepted, and legislation is in place to continue this process. 
It is a tribute to the state and schools to accept the risks and challenges which this type of 
reform, restructuring and systematic improvement requires each to face. From the 
researchers' observations, the process continues to be successful, and it is largely due to a 
continuing effort to provide decision makers with objective evaluation research evidence 
of accountability for attaining program goals. 
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APPENDIX A 



R Packand, L Bieriein, K Kundin 
PERCEPTION ASSESSMENT SCALE 

S II!! f'^^" below, please shsh the response which best describes the way you feel 

Career Lflr^flffr Program in VPlir f^p p rific district, not career ladder districts in general Do not resoonril? 
Items about which you may not have information. ^ ^° 

Rating Seals 

SA = Strongly Agree md = Moderately Disagree 

MA = I^oderately Agree SD = Strongly Disagree 

DNA = Does Not Apply 

General Ca reer Ladder r .nnf^ppjg- 

1 .1 The Career Ladder Program (CLP) will help attract high quality 

people into the teaching professton SA MA md SD DNA 

1 .2 The Career Ladder Program (CLP) will help retain the most 

competent tead-ers in the classroom ca ma n^n^ o,^ r..,. 

SA MA MD SD DNA 

1.3 The Career Ladder Program (CLP) will help improve instmction 



SA MA MD SD DNA 



1 .4 The Career Ladder Program (CLP) will help improve student 
academic progress. . 

^ SA MA MO SD DNA 



1 .5 The CLP encourages cooperation among teachers 

SA MA MD SD DNA 



SA MA MD SD DNA 

1.6 The CLP will tower teacher morale 



ERIC 



1 .7 The CLP will improve the professional status of teachers in the 
eyes of the public. ... 

SA MA MD SD DNA 

1 .8 The monetary rewards offered by the CLP are large enough of an 
incentivetocauseteacherstoappVfortheprogram SA MA MD SD DNA 

1 .9 The intrinsic rewards (personal satisfaction) are enough of an 

incentive forteachers to apply for the program SA MA MD SD DNA 

1.10 The district's career ladder goals and objectives have been 

cleaiiy communicated to teachers oa ..r, 

SA MA MD SD DNA 

1 .1 1 The CLP includes a fair appeal process for disagreements over 
placement on the ladder 

SA MA MD SD DNA 

2- Staff Pevelnnmenf and Jra jninn Onnrppt p 

2.1 ! have received adequate inservice on the CLP teacher 

evaluation system. ^. 

SA MA MD SD DNA 

2.2 Peer trainers (staff development personnel) have been selected 
onthebasisoftheirsuperiorqualificattons SA MA MD SD DNA 
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2.3 Administratorsare well trained in the CLP evaluation system SA MA MD SD DNA 

2.4 Credit earned in university courses should be one of the 

criteria for movement upward on the career ladder SA MA MD SD DNA 

2.5 The district provides adequate resources to help teachers 

gain the skills required for advancement on the ladder SA MA MD SD DNA 

3. Teacher Evaluation System Concepts 

3.1 The evaluation instruments clearly define 

the various levels of teaching performance SA MA MD SD DNA 

3.2 Teachers feel that administrators evaluate teaching performance 

fairiyfcrplacement on the ladder SA MA MD SD DNA 

3.3 The CLP evaluation procedures are stmctured in such a manner 

to insure consistency among evaluatore SA MA MD SD DNA 

3.4 The amount of time evaluators spend obsen/ing teachers is 

sufficient to ensure proper placements on the ladder, SA MA MD SD DNA 

3.5 Time required for the CLP evaluation process is worth the 

benefits gained SA MA MD SD DNA 

4. Peer Evaluation Concepts 

4.1 Peer evaluators have been selected on the basis of their superior 

qualifications SA MA MD SD DNA 

4.2 Peer evaluators are well trained in CLP evaluation procedures SA MA MD SD DNA 

4.3 Teachers have sufficient input in the selection of the peer evaluators 

involved in their evaluation SA MA MD SD DNA 

4.4 Peer evaluation is being used formatively (to assist teachers in the 

improvement of instruction) SA MA MD SD DNA 

4.5 Peer evaluation is being used summatively (to make decisions 

about placement in the CLP) SA MA MD SD DNA 

4.6 1 believe that peer evaluation in my district hinders cooperative 

staff efforts SA MA MD SD DNA 

5. Career Ladd er Placement Concepts 

5.1 Teachers clearly understand what is expected of them in order to 

advance on the ladder SA MA MD SD DNA 

5.2 Teachers can feel comfortable about choosing to remain at 

the same level on the ladder SA MA MD SD DNA 

5.3 The criteria for career ladder levels are challenging enough 




so that only the most competent teachers advance SA MA MD SD DNA 

5.4 The CLP clearly specifies standards for judging the contents of 

material submitted for CLP evaluation (portfolio, growth plan, etc.) SA ma MD SD DMA 

5.5 Adequate assistance is being provided to teachers regarding 

the development of materials submitted for CLP evaluation SA MA MD SD DMA 

5.6 Our CLP provides teachers with opportunities for continued 

advancement without leaving the classroom on a full-time basis SA MA MD SD DNA 

5.7 Teachers were adequately involved in the development of the 

district career ladder program SA MA MD SD DNA 

5.8 The positive effects of higher level responsibilities (teacher mentor, 
etc.) outweigh the possible disadvantages of being released part- ' 

time from classroom assignments SA MA MD SD DNA 

5.9 Clear criteria for CLP participation have been established for 
personnel whose job description differs from a regular classroom 

^^^^^ SA MA MD SD DNA 

5.10 Higher level responsibilities in the CLP are appropriate assignments 

for those teachers selected for advancement SA MA MD SD DNA 

5.1 1 The district has an adequate number of trained personnel to 

effectively place candidates on the career ladder SA MA MD SD DNA 

5.12 The district has established a means for adequate teacher 

input conceming possible revisions SA MA MD SD DNA 

(Feel free to respond on the back of this sheet ff more space is needed.) 

A. Please describe the major strength/s of your district career ladder program. 

B. Please describe the area/s of your career ladder program which need improvement. 

Career Larlrjvtr District ClimRtf? ^wmy The following questions are designed to assess teacher 
perceptions of general organizational climate. (Teacher evaluatorrj are asked to respond in relation 
to how you feel teachers will rate each item) 

6.1 I am treated with respect SA MA MD SD DNA 

6.2 I have a feeling of belonging SA MA MD SD DNA 

6.3 I have a feeling of being tmsted SA MA MD SD DNA 

6.4 I have feelings of being successful in my job assignment SA MA MD SD DNA 
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6.5 I am allowed to progress toward desired goals SA MA MD SD dna 

6.6 I have a feeling of being rewarded for a job well done SA ma MD sD dna 

6.7 I feel my work has a clear purpose SA MA MD SD DNA 

6.8 I feel my job has functional importance to the organization SA MA MD SD DNA 

6.9 I am consistently provided knowledge of progress SA MA MD SD DNA 

6.10 i feel I have ample options which provide hope for improvement SA MA MD SD DNA 

6.11 I am provided a cooperative woi'king environment SA MA MD SD dna 

6.12 I am not placed in unfair competitive situations with my peers SA MA MD SD dna 

6.13 I am provided good leadership models Sa MA MD CD dna 

6.14 I feel I am given a range of tasks which are challenging SA MA MD SD dna 

6.15 I feel my working area is pleasant SA MA MD SD DNA 

6.16 1 work in an environment free from excessive stress SA MA MD SD DNA 

6.17 lfeei;.-eefromfearinmywork SA MA MD SD DNA 

6.18 Organizational goals are clearly communicated SA MA MD SD DNA 

6.20 I feel appreciated by the supervisors who evaluate me SA MA MD SD DNA 

6.21 I am treated as an adult SA MA MD SD DNA 

6.22 My peers are treated with respect SA MA MD SD DNA 

6.23 f^y peers have feelings of being trusted SA MA MD SD DNA 

6.24 My peers have feelings of being rewarded for a job well done SA MA MD SD DNA 

6.25 My peers are consistently provided knowledge of their progress SA MA MD SD DNA 

6.26 My peers feel free from fear in their workplace SA MA MD SD DNA 

C. My district's greatest strength/s in the area of general climate is/are: 



D. My district's greatest need/s for improvement of general climate is/are: 
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